Handle EOF to prevent file descriptor leak #37

georgethebeatle · 2019-05-10T12:28:54Z

Currently the EOF that the server gets when a client connection is
closed is being ignored, which means that the goroutine that is
processing the client connection will never exit. If it never exits
it will never close the underlying unix socket and this would lead
to a file descriptor leak.

Signed-off-by: Georgi Sabev [email protected]

Reprodction steps:

# create a long running task
ctr image pull docker.io/library/alpine:latest
ctr run --rm docker.io/library/alpine:latest test sleep 9999

# count number of open sockets
ls -la /proc/$(pidof containerd-shim)/fd | grep socket | wc -l
2

# restart containerd to make it reconnect to the shim 
kill $(pidof containerd)
containerd & 

# observe that socket number increased. It increases by 1 with every containerd restart
ls -la /proc/$(pidof containerd-shim)/fd | grep socket | wc -l
3

Currently the EOF that the server gets when a client connection is closed is being ignored, which means that the goroutine that is processing the client connection will never exit. If it never exits it will never close the underlying unix socket and this would lead to a file descriptor leak. Signed-off-by: Georgi Sabev <[email protected]>

crosbymichael · 2019-05-10T13:33:13Z

LGTM

estesp

LGTM

stevvooe · 2019-05-10T16:41:42Z

server.go

@@ -452,6 +452,11 @@ func (c *serverConn) run(sctx context.Context) {
 			if err != nil && err != io.EOF {
 				logrus.WithError(err).Error("error receiving message")
 			}
+			if err == io.EOF || err == io.ErrUnexpectedEOF {


The err == io.ErrUnexpectedEO won't work unless you ignore it above.

It would not have worked if there was a return statement in the if above, but there isn't. It would still be better to not log the ErrUnecpectedEOF though so we have created a new pr: #38. Pls, take a look.

From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7 > Welcome to the v1.2.7 release of containerd! > > The seventh patch release for containerd 1.2 introduces OCI image > descriptor annotation support and contains fixes for containerd shim logs, > container stop/deletion, cri plugin and selinux. > > It also contains several important bug fixes for goroutine and file > descriptor leakage in containerd and containerd shims. > > Notable Updates > > - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254 > - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31 > - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37 > - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264 > - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273 > - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby#38978 > - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282 > - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283 > - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177 > - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159 > - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50 > - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316 > - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330 Signed-off-by: Sebastiaan van Stijn <[email protected]>

From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7 > Welcome to the v1.2.7 release of containerd! > > The seventh patch release for containerd 1.2 introduces OCI image > descriptor annotation support and contains fixes for containerd shim logs, > container stop/deletion, cri plugin and selinux. > > It also contains several important bug fixes for goroutine and file > descriptor leakage in containerd and containerd shims. > > Notable Updates > > - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254 > - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31 > - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37 > - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264 > - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273 > - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978 > - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282 > - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283 > - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177 > - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159 > - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50 > - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316 > - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330 Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: d5669ec1c6eedcd5dd8b0ecd615638934561daa4 Component: engine

full diff: containerd/ttrpc@699c4e4...1fb3814 changes included: - containerd/ttrpc#37 Handle EOF to prevent file descriptor leak - containerd/ttrpc#38 Improve connection error handling - containerd/ttrpc#40 Support headers - containerd/ttrpc#41 Add client and server unary interceptors - containerd/ttrpc#42 Refactor close handling for ttrpc clients - containerd/ttrpc#43 metadata as KeyValue type Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: containerd/ttrpc@699c4e4...92c8520 changes: - containerd/ttrpc#37 Handle EOF to prevent file descriptor leak - containerd/ttrpc#38 Improve connection error handling - containerd/ttrpc#40 Support headers - containerd/ttrpc#41 Add client and server unary interceptors - containerd/ttrpc#43 metadata as KeyValue type - containerd/ttrpc#42 Refactor close handling for ttrpc clients - containerd/ttrpc#44 Fix method full name generation - containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up) - containerd/ttrpc#49 Handle ok status Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: containerd/ttrpc@699c4e4...92c8520 changes: - containerd/ttrpc#37 Handle EOF to prevent file descriptor leak - containerd/ttrpc#38 Improve connection error handling - containerd/ttrpc#40 Support headers - containerd/ttrpc#41 Add client and server unary interceptors - containerd/ttrpc#43 metadata as KeyValue type - containerd/ttrpc#42 Refactor close handling for ttrpc clients - containerd/ttrpc#44 Fix method full name generation - containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up) - containerd/ttrpc#49 Handle ok status Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: 8769255d1bb9c469d4f2966e7e9869a9f126f9e9 Component: engine

From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7 > Welcome to the v1.2.7 release of containerd! > > The seventh patch release for containerd 1.2 introduces OCI image > descriptor annotation support and contains fixes for containerd shim logs, > container stop/deletion, cri plugin and selinux. > > It also contains several important bug fixes for goroutine and file > descriptor leakage in containerd and containerd shims. > > Notable Updates > > - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254 > - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31 > - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37 > - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264 > - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273 > - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby#38978 > - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282 > - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283 > - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177 > - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159 > - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50 > - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316 > - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330 Signed-off-by: Sebastiaan van Stijn <[email protected]> (cherry picked from commit d5669ec) Signed-off-by: Sebastiaan van Stijn <[email protected]>

full diff: containerd/ttrpc@699c4e4...92c8520 changes: - containerd/ttrpc#37 Handle EOF to prevent file descriptor leak - containerd/ttrpc#38 Improve connection error handling - containerd/ttrpc#40 Support headers - containerd/ttrpc#41 Add client and server unary interceptors - containerd/ttrpc#43 metadata as KeyValue type - containerd/ttrpc#42 Refactor close handling for ttrpc clients - containerd/ttrpc#44 Fix method full name generation - containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up) - containerd/ttrpc#49 Handle ok status Signed-off-by: Sebastiaan van Stijn <[email protected]> (cherry picked from commit 8769255) Signed-off-by: Sebastiaan van Stijn <[email protected]>

From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7 > Welcome to the v1.2.7 release of containerd! > > The seventh patch release for containerd 1.2 introduces OCI image > descriptor annotation support and contains fixes for containerd shim logs, > container stop/deletion, cri plugin and selinux. > > It also contains several important bug fixes for goroutine and file > descriptor leakage in containerd and containerd shims. > > Notable Updates > > - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254 > - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31 > - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37 > - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264 > - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273 > - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978 > - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282 > - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283 > - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177 > - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159 > - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50 > - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316 > - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330 Signed-off-by: Sebastiaan van Stijn <[email protected]> (cherry picked from commit d5669ec1c6eedcd5dd8b0ecd615638934561daa4) Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: 768923199f89246ff51039ae030e4b492f8d4555 Component: engine

full diff: containerd/ttrpc@699c4e4...92c8520 changes: - containerd/ttrpc#37 Handle EOF to prevent file descriptor leak - containerd/ttrpc#38 Improve connection error handling - containerd/ttrpc#40 Support headers - containerd/ttrpc#41 Add client and server unary interceptors - containerd/ttrpc#43 metadata as KeyValue type - containerd/ttrpc#42 Refactor close handling for ttrpc clients - containerd/ttrpc#44 Fix method full name generation - containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up) - containerd/ttrpc#49 Handle ok status Signed-off-by: Sebastiaan van Stijn <[email protected]> (cherry picked from commit 8769255d1bb9c469d4f2966e7e9869a9f126f9e9) Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: 525e8ed3febff46d07cb01961601824b5f8b301b Component: engine

From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7 > Welcome to the v1.2.7 release of containerd! > > The seventh patch release for containerd 1.2 introduces OCI image > descriptor annotation support and contains fixes for containerd shim logs, > container stop/deletion, cri plugin and selinux. > > It also contains several important bug fixes for goroutine and file > descriptor leakage in containerd and containerd shims. > > Notable Updates > > - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254 > - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31 > - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37 > - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264 > - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273 > - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby#38978 > - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282 > - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283 > - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177 > - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159 > - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50 > - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316 > - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330 Signed-off-by: Sebastiaan van Stijn <[email protected]> (cherry picked from commit d5669ec) Signed-off-by: Sebastiaan van Stijn <[email protected]>

From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7 > Welcome to the v1.2.7 release of containerd! > > The seventh patch release for containerd 1.2 introduces OCI image > descriptor annotation support and contains fixes for containerd shim logs, > container stop/deletion, cri plugin and selinux. > > It also contains several important bug fixes for goroutine and file > descriptor leakage in containerd and containerd shims. > > Notable Updates > > - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254 > - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31 > - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37 > - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264 > - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273 > - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978 > - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282 > - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283 > - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177 > - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159 > - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50 > - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316 > - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330 Signed-off-by: Sebastiaan van Stijn <[email protected]> (cherry picked from commit d5669ec1c6eedcd5dd8b0ecd615638934561daa4) Signed-off-by: Sebastiaan van Stijn <[email protected]> Upstream-commit: 8c7928adaa83264947b6e296eeb068b99843822e Component: engine

From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7 > Welcome to the v1.2.7 release of containerd! > > The seventh patch release for containerd 1.2 introduces OCI image > descriptor annotation support and contains fixes for containerd shim logs, > container stop/deletion, cri plugin and selinux. > > It also contains several important bug fixes for goroutine and file > descriptor leakage in containerd and containerd shims. > > Notable Updates > > - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254 > - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31 > - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37 > - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264 > - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273 > - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby#38978 > - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282 > - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283 > - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177 > - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159 > - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50 > - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316 > - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330 Signed-off-by: Sebastiaan van Stijn <[email protected]> Signed-off-by: zach <[email protected]>

full diff: containerd/ttrpc@699c4e4...92c8520 changes: - containerd/ttrpc#37 Handle EOF to prevent file descriptor leak - containerd/ttrpc#38 Improve connection error handling - containerd/ttrpc#40 Support headers - containerd/ttrpc#41 Add client and server unary interceptors - containerd/ttrpc#43 metadata as KeyValue type - containerd/ttrpc#42 Refactor close handling for ttrpc clients - containerd/ttrpc#44 Fix method full name generation - containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up) - containerd/ttrpc#49 Handle ok status Signed-off-by: Sebastiaan van Stijn <[email protected]> Signed-off-by: zach <[email protected]>

georgethebeatle force-pushed the pr-handle-client-eof branch from 7f90d4b to 96dcf73 Compare May 10, 2019 12:35

estesp approved these changes May 10, 2019

View reviewed changes

estesp merged commit 69eadd1 into containerd:master May 10, 2019

stevvooe reviewed May 10, 2019

View reviewed changes

This was referenced May 13, 2019

Improve connection error handling #38

Merged

Bump ttrpc containerd/containerd#3278

Merged

thaJeztah mentioned this pull request Jun 14, 2019

Update containerd to v1.2.7 moby/moby#39364

Merged

This was referenced Jun 14, 2019

[19.03 backport] Update containerd to v1.2.8 docker-archive/engine#282

Merged

[18.09 backport] Update containerd to v1.2.8 docker-archive/engine#283

Merged

thaJeztah mentioned this pull request Aug 9, 2019

bump containerd and dependencies to v1.3.0 moby/moby#39713

Merged

thaJeztah mentioned this pull request Aug 27, 2019

bump google.golang.org/grpc v1.23.0 (CVE-2019-9512, CVE-2019-9514, CVE-2019-9515) moby/moby#39798

Merged

thaJeztah mentioned this pull request Aug 28, 2019

[19.03 backport] bump google.golang.org/grpc v1.23.0 (CVE-2019-9512, CVE-2019-9514, CVE-2019-9515) docker-archive/engine#340

Merged

thaJeztah mentioned this pull request Sep 6, 2019

[19.03 backport] Update containerd to v1.2.9 docker-archive/engine#349

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle EOF to prevent file descriptor leak #37

Handle EOF to prevent file descriptor leak #37

georgethebeatle commented May 10, 2019 •

edited

Loading

crosbymichael commented May 10, 2019

estesp left a comment

stevvooe May 10, 2019

georgethebeatle May 13, 2019

Handle EOF to prevent file descriptor leak #37

Handle EOF to prevent file descriptor leak #37

Conversation

georgethebeatle commented May 10, 2019 • edited Loading

crosbymichael commented May 10, 2019

estesp left a comment

Choose a reason for hiding this comment

stevvooe May 10, 2019

Choose a reason for hiding this comment

georgethebeatle May 13, 2019

Choose a reason for hiding this comment

georgethebeatle commented May 10, 2019 •

edited

Loading